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BACKGROUND COMMENTS 


This report documents the work of one component of a larger research effort. The objective of the 
project is to understand what schools of education are doing to prepare teachers to use data in their 
practice. The issue is multifaceted, complex, and systemic. Schools of education do not act alone to 
suddenly introduce courses on data-driven decision making into their curricula because they have a 
whim to do so or because policymakers say that data literacy among educators is important. Schools 
of education must come to realize on their own that building the human capacity to use data among 
their teacher candidates is a response to needs from the field, stimulated in part by policymakers’ 
rhetoric that education must become an evidence-based field. The study, of which this report is a 
part, contains three distinct, but interconnected components that, in combination, provide a 
depiction of the landscape of teacher preparation and data literacy. The components include a 
survey to schools of education, a review of selected syllabi, and an analysis of state licensure 
documents and requirements. This document focuses on the licensure component. 


RELEVANT LITERATURE 


The field has struggled to define what it means to be a data literate educator (Mandinach & 
Gummer, 2011, 2013a). We brought together 55 experts in the field, with the specified objective of 
achieving a common definition. Yet the best we were able to achieve was roughly 95 percent 
agreement, with the remaining 5 percent rather amorphous. Others have posited variations on 
definitions (Data Quality Campaign Data Literacy Group, 2013; North Carolina Department of 
Public Instruction, 2013). In terms of applicability specifically for teaching, it is our belief that the 
following definition approaches the construct in the most comprehensive manner possible: 

Pedagogical data literacy or data literacy for teaching is the ability to transform information 
into actionable instructional knowledge and practices by collecting, analyzing, and 
interpreting all types of data (assessment, school climate, behavioral, snapshot, longitudinal, 
moment-to-momemt, etc.) to help determine instructional steps. It combines an 
understanding of data with standards, disciplinary knowledge and practices, curricular 
knowledge, pedagogical content knowledge, and an understanding of how children learn. 


WHAT DO POLICY AND RESEARCH SAY? 

Much attention from policymakers has been given to the importance of teachers using data. 
Secretary of Education Arne Duncan (2009a, 2009b, 2009c, 2010a, 2010b, 2012) has spoken widely 
about the need for teachers to use data and the importance of such evidence-driven practice. In 
fact, Duncan (2012) publically challenged schools of education to step up and begin to train 
educators at a national conference sponsored by the Data Quality Campaign. Further, data use is 
one of the four pillars in the American Recovery and Reinvestment Act (ARRA, 2009) and in the 
Race to the Top (U.S. Department of Education, 2009). 

Professional organizations such as the National Council for Accreditation of Teacher Education 
(NCATE) and the Council of Chief State School Officers (CCSS0) have included data literacy or the 
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capacity to use data among their recommendations and standards. The National Board of 
Professional Teaching Standards also has been a strong proponent of improving teachers’ capacity 
to use data (Aquerrebere, 2009). A Blue Ribbon Panel (2010) report released by NCATE and 
endorsed by Duncan (2010b) recommended that teacher candidates know how to make decisions. 

It further recommends that teacher candidates must be able to analyze student learning needs and 
make instructional adjustments by using student performance data and other sources of data to 
inform their practice. 

CCSSO (2011) released the InTASC standards for teaching that laid out 10 recommendations, each 
with knowledge, dispositions, and performance skills that are required of teachers. The document 
identifies “using data to support learning” as one of the cross-cutting themes. It further specifies 
that the data theme occurs in 43 of the knowledge, dispositions, and performance components. We 
analyzed the document further and noted an additional 24 components. Suffice it to say that the 
components of data literacy are well represented in the InTASC standards. 

In some ways, policymakers are further along in their thinking about data literacy among educators 
than are researchers. Policymakers and researchers in the area of data-driven decision making have 
focused on teachers in a number of ways, but has rarely addressed teacher preparation. Many 
articles and studies have noted the importance for teachers to know how to use data effectively to 
inform their practice and the need to build educators’ capacity to use data (Baker, 2003; Choppin, 
2002; Feldman & Tung, 2005; Hamilton, Halverson, Jackson, Mandinach, Supovitz, & Wayman, 
2009; Ikemoto & Marsh, 2007; Mandinach, 2009, 2012; Mandinach & Honey, 2008; Mason, 2002; 
Miller, 2009). There have been numerous calls for high-quality and sustained professional 
development to facilitate data literacy (Baker, 2003; Mandinach, Rivas, Light, & Heinze, 2006; 
Means, Padilla, & Gallagher, 2010; Schafer & Lissitz, 1987; Wise, Lukin, & Roos, 1991). Yet 
preparing educators to use data only goes so far. Having good professional development is 
important, but there also is a pressing need for the infrastructure to support the infusion of data use 
into schools and districts (Marsh, Pane, & Hamilton, 2006). 

While the existing literature focuses on the current cohort of teachers, in-service training, and 
professional development, little, if any attention has been devoted to teacher preparation and pre¬ 
service. Recognizing the dearth of knowledge about the role of teacher preparation in developing 
data literacy, early in 2011, we convened a meeting of key stakeholders to discuss what schools of 
education can do to prepare educators to use data 1 . The outcome of that meeting was a white paper 
(Mandinach & Gummer, 2011) and a call to action that appeared in the Educational Kesearcher 
(Mandinach & Gummer, 2013b). The white paper reported on the varying perspectives of different 
stakeholder groups, as well as a clear picture of many of the challenges. It outlined a research 
agenda needed to inform the field, including a comprehensive survey to schools of education to 
better understand the landscape of course offerings. The journal article laid out the systemic nature 
of the problem and took the perspective that professional development providers can only go so far 
as to train some of the current cohort of teachers. It was clear that something must be done to 
improve the pipeline of educators, looking to schools of education to respond by integrating data 
use into their course offerings to address the need at the pre-service level. 


1 This work was sponsored by the Spencer Foundation. 
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PROJECT DESCRIPTION: INTEGRATING TWO STUDIES TO UNDERSTAND THE 
LICENSURE REQUIREMENTS 


This report is formally about the examination of state licensure requirements for teacher candidates. 
This examination provides only part of the needed information by which to understand, interpret, 
and draw conclusions about such documentation. The requirements must necessarily be placed 
within the context of data literacy and the skills that comprise that construct. Therefore to 
contextualize the current findings, we also report on a prior study that provides findings about what 
experts in the field of data-driven decision making report as key skills and knowledge Placing the 
expert findings side-by-side with the licensure requirements enables us to identify commonalities, 
differences, and gaps. We describe both projects’ methods and analytic approaches. We then align 
the two studies to examine trends and differences. 


METHODS 

The methods from this work emanate from two separate studies. The first study focused on 
defining data literacy (Mandinach & Gummer, 2011, 2013a). The second study examined what 
schools of education are doing to build data literacy among teacher candidates and current teachers 
(Mandinach, Gummer, & Friedman, in press). 

The Prior Study 

In the data literacy study, 55 experts from diverse perspectives (researchers, professional 
development providers, professors, funders, and policymakers in data-driven decision making and 
formative assessment) were convened and asked to define data literacy. Each expert submitted a 
definition. The resulting definitions were analyzed to identify particular skills and knowledge that 
the experts reported educators must have to use data effectively. Transcripts from the two-day 
conference also were analyzed for evidence of specific skills and knowledge. The process yielded 
nearly 100 skills that initially categorized into: problem focus; data focus; process focus; and 
disciplinary, topical, dispositional, and other knowledge. 

Since the convening and the analysis of the findings, additional work has been conducted to develop 
a conceptual framework for data literacy for teaching. This work has entailed a more intensive 
examination and analysis of the skills and knowledge, with an explicit attempt to identify and further 
refine the constructs and categorize the skills as part of the constructs. The framework development 
process also includes identifying interrelationships among the constructs, using concept mapping 
procedures. 

The Current Study 

The current study, with the focus on what schools of education are doing to build educators’ 
capacity to use data, includes three different analytic components. It includes a survey to schools of 
education, an analysis of syllabi for data courses of related courses, and an examination of states’ 
licensure documents. The component that pertains to this article is the licensure work. 

With the assistance of the president of the National Association of State Directors of Teacher 
Education Credentialing (NASDTEC), we sought to identify and locate documents on states’ 
websites that outline the specific skills and knowledge that individuals seeking licensure would need 
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to exhibit. This is a challenging task for several reasons. Location is a first issue. Sometimes the 
documents reside on the state education agency’s website, while other times, they might reside in 
associated agencies’ website. Content is a second issue. Some states do not include any details in 
terms of specific skills and knowledge that are required. Focus is a third issue. Some documents 
focus more on program requirements than on what candidates need to know. Specificity is a fourth 
issue. Some documents are cursory at best, whereas others are detailed to the level of licensure 
within specific content areas and grade levels. The documents ranged from less than a page to 
several hundred pages. Some states had multiple documents, whereas finding something relevant 
for other states was a real challenge. Fifth, some states abdicate to specific standards and rubrics, 
for example the InTASC standards (CCSSO, 2011) or the Danielson framework (Danielson, 2013). 

The documents were searched first for the term “data” and then for “assessment”. In instances 
where those terms were found, the statements were highlighted and logged. Skills and knowledge 
were noted, as were outcomes or trends. For example, “teachers analyze data to monitor student 
progress, learning, and to plan, differentiate and modify instruction.” “Analyze”, “monitor”, 
“plan”, “differentiate”, and “modify” would be considered skills. “Student learning progress” and 
“instruction” would be outcomes. Or, ” to collect, synthesize, and evaluate data for the purpose of 
continuous improvement.” In this example, “collect” “analyze”, and “evaluate” would be skills and 
“continuous improvement” would be the outcome. Or, “collect, analyze, and interpret formal 
assessment data to report all stakeholders using effective communication skills.” The skills noted 
here would include: “collect”, “analyze”, “interpret”, “assessment”,” involvement of stakeholders”, 
and “communicate”. In addition to the targeted searches, each document was visually scanned for 
other statements that would pertain to data use because relevant indices did not always contain the 
two searched terms. For example, “teachers collaborate and communicate student progress with 
students, parents, and colleagues.” In this instance, the skills would be “collaborate”, 
“communicate”, and involve stakeholders. The outcome would be “student progress”. 

A rubric and several matrices were constructed. The rubric consisted of several overarching 
variables, including: 

• The amount of information about data contained in the documents; 

• Whether the term and concept of “data” was included; 

• Whether the term and concept of “assessment” was included; 

• Whether specific skills were stated; 

• The specificity of the documentation in terms of data-related concepts; 

• If the state uses the InTASC (CCSSO, 2011) standards; 

• If there is a developmental continuum for the skills extending from novice to expert 
teacher candidates; 

• Whether there is a specific data strand; 

• If the state uses the Danielson (2013) framework; and 

• A judgment about if the documents pertain to data literacy or assessment literacy. 

These variables were applied to each state’s documentation. Thus, a state by variable matrix was 
constructed. Another matrix was developed that crossed the specific skills and knowledge by state. 
A final matrix included the outcomes categorized by state. 
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The skills from both studies were combined into a comprehensive list. Roughly 100 skills were 
noted. Two experts examined the list for commonalities among the skills and knowledge. The skills 
were categorized into components and constructs. Redundant skills, due to different but analogous 
terminology, were removed. The end result was a list of 58 unique skills, including three 
superordinate skills. The categorization scheme also noted the genesis of each skill; that is, whether 
it came from the expert definitions, the licensure documents, or both. 


RESULTS 

We were able to locate documents for all states so our data included all 50 states and the District of 
Columbia. Six states (AR, AZ, DC, NV, ND, and SC) abide by the InTASC standards (CCSSO, 
2011) and one state uses the Danielson framework (2013). All other states have some sort of 
documentation, that, more or less, addresses what knowledge and skills teacher candidates need in 
order to be licensed. 

General Categorization 

The documents were rated for several variables that reflect the integration of data use concepts. A 
first rating focused on the extent to which a state explicitly focused on data. This is important to 
determine because of the existing confusion between data literacy and assessment literacy among 
policymakers, practitioners, and some researchers (Mandinach & Gummer, 2011, 2013a). Twelve 
states fail to include any mention of data in their documentation, while 9 states barely mention data, 
and 30 states explicitly deal with the concept (see Appendix A). In contrast, only two states (MT, 
WY) do not mention assessment in their regulations. All but seven states describe specific skills that 
teacher candidates must exhibit. However, there is a wide range in the degree of specificity of the 
knowledge and skills listed. Some states such as Ohio, South Carolina, and Virginia include highly 
specific statements about requisite skills and knowledge, whereas several states (MS, MT, WI, MA, 
WA, and NE) totally lack specificity. 

The amount of information about data and assessment also varied. Some states barely included 
anything relevant (NE, WA, MT, MS, and WI), whereas others were steeped in information (OH, 
VA, and the InTASC states). Seven states (MI, NM, WV, SD, KY, OH, GA) have developmental 
continua that outline how the skills differ from unacceptable or novice, to expert. Eight states have 
a specific data standard, but even these differ. For example, the District of Columbia has a standard 
entitled, “Data and Assessment: yet the standard barely mentions data. In contract, North Carolina 
has an entire document devoted to describing data-driven decision making and data literacy (North 
Carolina Department of Public Instruction, 2013): 

Data literacy refers to one’s level of understanding of how to find, evaluate, and use data to 
inform instruction”. It further states, “a data literate person possesses the knowledge to 
gather, analyze, and graphically convey information and data to support decision-making, (p. 

i) 

The documents were rated to determine if they deal with data literacy and assessment literacy. 

Under half (21) address data literacy. However, states like North Carolina, Ohio, South Carolina, 
Utah, Virginia, and Alabama do an outstanding job of outlining the skills that comprise data literacy. 
More states (37) address assessment literacy with five (SC, UT, OH, HI, IL) specifically focusing on 
the construct. It is not unexpected that more states deal with assessment literacy than data literacy 
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because of the nature of the data with which teachers typically deal. What is interesting is finding 
the relative emphasis heavily toward data literacy with no mention of assessment literacy in states 
like Alabama, North Carolina, Ohio, and Tennessee. 

Identifying the Specific Skills 

The experts’ definitional study yielded nearly 100 skills and knowledge topics around data use 
(Mandinach & Gummer, 2011, 2013a). As noted above, they were categorized into: (a) problem 
focus; (b) data focus; (c) process focus; and (d) disciplinary, topical, dispositional, and other 
knowledge. Interestingly, some experts recognized that for teachers, there is a need to integrate 
pedagogy and content knowledge with data analytics. Content knowledge is not a component found 
among other frameworks for data use (Mandinach & Jackson, 2012). As we touch upon later and 
discuss at length elsewhere (Gummer & Mandinach, in press), our conceptual framework, what we 
refer to as data literacy for teaching, consists of data literacy skills (the typical analytic aspects), 
content knowledge, and pedagogical content knowledge. The later two constructs adapt data literacy 
to the specific context of teaching and instruction. 

Appendix B provides a list of the 70 skills and knowledge that remain from the experts’ definitions. 
Note that some of the variables are complex, having combined similar skills. Also of interest, but 
not reflected in the list, is that some experts recognized a developmental continuum for data use 
more generally, not specific skills. They postulated and discussed how basic data literacy might look 
in contrast to full fluency or expertise in data literacy. Are they the same skills, but with increasing 
sophistication? Are they different subsets of skills? To what extent do the skills differ or evolve 
based on one’s role, level of experience, and expertise? 

The licensure analyses enabled us to identify specific skills for individual states to see if there was 
evidence of inclusion of data literacy in licensure requirements. Assessment literacy was also 
included in the analyses because it is seen as a component of data literacy (Mandinach & Gummer, 
2011,2013a). 


States varied tremendously in the number of skills found in their documentation, ranging from none 
(WY) and three (AK) to 52 (ND and SC). The states with the most skills were those that use the 
InTASC standards, although for some of these states, additional skills were identified. The average 
number of skills per state was 21.3 with a standard deviation of 13.82. There were 15 states with 0 
to 10 skills, 12 with 11 to 20, 14 with 21 to 30, and 10 with 31 or more. The six states with the most 
skills were the InTASC states followed by Illinois (36), Alabama (35), Kansas (34), and New 
Hampshire (34). 


The number of states that identified specific skills also varied greatly, from two states (for 
manipulate and infer), to a high of 43 (for assess). The average number of states per skills was 18.11 
with a standard deviation of 11.06. Skills noted by over 20 of the states included: 


Plan (39) 

Use data (37) 
Monitor (36) 
Analyze (33) 
Collect/gather (32) 
Collaborate (28) 
Modify (28) 
Interpret (26) 
Reflect (26) 


Use multiple measures (39) 
Involve stakeholders (36) 
Communicate (34) 
Evaluate (33) 
Document/review (31) 
Make decisions (28) 
Provide feedback (27) 
Adjust (26) 

Adjust (26) 
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Use technology (25) 
Use different data (23) 
Differentiate (21) 


Design/guide (26) 
Examine (24) 
Adapt (21) 
Identify/select (20) 


Some 

noted 


skills that the literature and experts acknowledge as being important to effective data use were 
by fewer states. They include: 

Access/retrieve/find (13) 

Use displays and representations (14) 

Organize data (12) 

Draw conclusions (11) 

Use statistics (8) 


Infer (7) 

Integrate (6) 
Predict/hypothesize (5) 


Use of inquiry (15) 
Synthesize (13) 

Data quality (12) 

Patterns and trends (9) 
Examine data (8) 
Challenge assumptions (6) 
Verify (6) 

Summarize (4) 


It is interesting that 19 states included the use of research and 18 considered the ethical use of data 
as important. These are topics of increasing importance 


When we examined the documents for outcomes, a somewhat expected pattern emerged. Of the 20 
outcomes identified, the two most frequent uses of data are for assessment (48) and instruction (46), 
followed by student progress (41) and student learning (40). Other uses included to inform 
performance (34), curricular goals (28), meet student needs (32), increase growth (20), and to inform 
teaching (22). The least frequent uses were for program effectiveness (6), accountability (7), learning 
gains (7), guidance (8), readiness (9), and patterns and trends (9). The average number of states per 
use was 13.97 with a standard deviation of 21.8. 


Alignment and Component Identification 

The skills and knowledge generated from the two studies were combined into one list. The origin of 
each skill was noted; that is, did it come from the experts’ definitions, the licensure documents, or 
both. An attempt was made to reduce redundancy but combining similar skills. The result was a list 
of 58 skills, including three superordinate skills (see Appendix C). The experts generated 50 skills 
and the licensure documents yielded 44 skills. Of the 58 skills, 35 were noted in both sources. The 
nine skills that were unique to the licensure documents pertained to the application of data to the 
instructional process. For example, planning, designing and guiding instruction, individualizing, and 
differentiating definitely are part of pedagogy and instruction. Some were considered procedural, 
such as reflecting on data, reviewing, and documenting. Skills unique to the experts pertained to 
general data-driven decision making. For example, licensure does not address the need to identify 
problems of practice, how to frame questions, understanding context, causality, testing assumptions, 
and generating and testing hypotheses. Also excluded are some issues around data quality, using 
qualitative data, and considering the consequences of a decision. 

The next step in the analytic process was to categorize and name related components and processes. 
This was an essential step in the development of our conceptual framework. Three superordinate or 
overarching components were identified. They are: (a) engage in an inquiry cycle; (b) transforming 
data to information to actionable knowledge; and (c) communication. Data-driven decision making 
is seen as a cycle or process of inquiry. The cycle pervades all that goes on during decision making. 
The process also is a continuum of transforming data into actionable knowledge that can be applied 
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to an educational problem, issue, or question. Communication is required, whether talking to 
colleagues, students, parents, or other stakeholders. Eight components were identified: (a) inquiry 
processes; (b) habits of mind; (c) general data use; (d) data quality; (e) data properties; (f) 
transformation of data to interpretation; and (g) transformation of data to interpretation. 

Inquiry Processes 

Skills that are part of the inquiry process focus on aspect of identifying a problem, an issue, or a 
question, thinking critically, and understanding the context of the problem. Ten such skills were 
identified. Teachers should be able to: 

• Identify a problem ofpractice about a student, group of students, a topical area, the 
curriculum, or an aspect of instruction. (Only from the experts’ definitions.) 

• Frame a question that pertains to the particular problem or issue. (Only from the 
experts’ definitions.) 

• Understand the context for the decision to concretize the situation, the needs, and the 
environmental situation. (Only from the experts’ definitions.) 

• Involve stakeholders , including students, so that the right resources are applied to the 
query; that is, students, parents, other teachers, and educators should be consulted to 
weigh in on the issue at hand. 

• Generate hypotheses about why the particular problem may be occurring and what 
might be done to remediate it. The inquiry should be informed by hypotheses to 
narrow the universe of possibilities. 

• Probe for causality , as an attempt to understand why the behavior, performance, or 
situation has happened. (Only from the experts’ definitions.) 

• Test assumptions early in the inquiry cycle to help determine if you are on the right track 
or off base. (Only from the experts’ definitions.) 

• Fvaluate outcomes, scrutinize results, and engage a feedback loop form the cycle of inquiry. 

• Determine next steps, because an inquiry an inquiry cycle is not linear, but rather 
iterative. (Only from the experts’ definitions.) 

• Test hypotheses later in the inquiry cycle to determine if the decision making process is 
leading to a successful conclusion. (Only from the experts’ definitions.) 

Habits of Mind 

Habits of mind are general strategies, dispositions, or tools that surround and influence the use of 
data. Three skills were identified: 

• Think critically about the problem at hand while using the inquiry cycle to use data to 
inform a decision. (Only from the experts’ definitions.) 

• belief in data is seen as an important facilitating factor in effective data use. If 
educators do not believe that data can inform practice or are skeptical about such 
use, they are less likely to use data or evidence in their work. (Only from the experts’ 
definitions.) 

• Collaboration is considered an important and valued component in the inquiry process 
where educators work together to examine data and seek solutions to a particular 
problem. (Only from the experts’ definitions.) 

General Data Use 

Two skills were categorized under general data use because of their global nature. 
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• Use data is a term that was noted by both experts and the in the licensure documents 
as a general statement concerning the importance of educators understanding why 
data are to be used and how to use them to inform their practice. 

• Ethics of data use is sometimes overlooked as an important component in data literacy 
but understanding the appropriateness of why and how data are to be use is indeed 
an essential skill. 

Data Quality 

Data quality is a topic on which the experts focused more because it is a general principal of data- 
driven decision making, and not necessarily something that is readily taught in teacher preparation 
programs. 

• Understanding data quality — accuracy, completeness is an essential component of using data 
effectively. Foundational to data use is knowing that the data being used are “clean”, 
that is, accurate and complete. 

• Understanding problematic data is a skill that involves being able to recognize when there 
is an issue with the data, whether a data point is out of range, inaccurate, or 
incomplete. (Only from the experts’ definitions.) 

• Knowledge of assessments and their psychometric properties refers to the understanding of 
concepts such as reliability, validity, and error of measurement. 

Data Properties 

The properties of the data refer to the alignment of the data to the purpose of the inquiry; that is, 
making sure that the appropriate data are being used to solve the problem. More specifically to 
assessment, it is the knowledge of what measures yield what kinds of data and their alignment to the 
question at hand. 

• Understanding the purpose of different data because different data have different uses and 
utility. 

• Identify and evaluate the right data for the particular problem of interest. 

• Use qualitative data. It is important to recognize that not all data are quantitative and 
that qualitative indices can be valued and informative sources. (Only from the 
experts’ definitions.) 

• Use quantitative data. Much of the data used in education is considered quantitative 
(such as assessments) and therefore knowing how to use such data is essential. 

• Understanding what data are not applicable ; that is, not all data are appropriate or 
applicable for every given circumstance. It is important to recognize when specific 
data are or are not relevant for the problem at hand. 

• Use formative and summative assessments and knowing when which sources of data are 
appropriate is a key skill for teachers. These sources of data yield different kinds of 
information that may be more or less aligned to instructional practice. 

• Use multiple measures / sources of data is a foundational concept in data-driven decision 
making and educational measurement; that is, it is important not to rely on just one 
measure but to triangulate among multiple sources of data to obtain a better and 
more accurate depiction of the situation. 

• Use research evidence to inform educational practice, taking results from the literature 
and applying them in actual educational settings. 
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Data Use Procedural Skills 

The procedural skills are the actions taken to attain a decision. They are the knowing “how” to use 
data or the “what” you do with data skills. These skills are the fundamentals of data literacy. 

• Use technologies to support data use through data warehouses, assessment systems, 
student information systems, instructional management systems, data dashboards, 
simple spreadsheets, and other relevant technologies that provide access to, analysis, 
and reporting of data. 

• Find, locate> access , and retrieve data in data systems, grade books, Excel files, and other 
storage locations. Increasingly, educational data are stored in electronic formats for 
easy and safe access and analysis. 

• Generate data because not all data are already produced. Teachers generate a plethora 
of data every day, ranging from moment to moment assessments of students to more 
long-term determinations of students’ understanding. 

• Collect , gather ; and store data means pulling together the data from tests, reports, 
portfolios, and other activities, coding and archiving the information in ways that are 
readily retrievable from electronic files or even paper files. 

• Organise data into a meaningful and manageable representation of the information. 

• Prioritise data because not all data are as relevant or as important as others. It is 
therefore important to arrange the data according to the utility of the issue being 
addressed. 

• Manipulate data entails handling or treating the data as part of the examination 
process. 

• Examine data means to scrutinize or inspect them in a meaningful way to address a 
particular question, hypothesis, or issue. (Only from the licensure documents.) 

• Manage data because there is such a multitude of data, that the wealth of data must be 
handled with accuracy, coded, stored, and arranged in a coherent manner for latter 
access and examination. (Only from the licensure documents.) 

• Reflect on data or consider the data systematically using an hypothesis to guide the 
process. (Only from the licensure documents.) 

• Process data by preparing them for examination and analysis. (Only from the licensure 
documents.) 

• Integrate data by examining, analyzing, and combining them in some meaningful way 
for sense making. (Only from the licensure documents.) 

• Review and document the findings from the examination of the data in a way that 
effectively communicates the results. (Only from the licensure documents.) 

• Develop sound assessments whether by hand or through the use of assessment systems. 
This skill entails knowing how to design items and combine them into a meaningful 
test that yields reliable and valid data. 

• Drilling down , aggregating, disaggregating, strand\ items are all functions of examining 
various representations and forms of assessment data outputs. Total test scores do 
not tell the complete story. Teachers need to drill down to strand or item levels. 
They must be able to examine aggregate and disaggregate levels of data. 

Transformation of Data to Information 

The transformation of data into information is the first step in working toward the provision of 
actionable knowledge as a continuum of steps of using data to inform practice. The skills that 
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comprise this component focus on the examination and analysis of data in ways that lead toward 
interpretable results. 

• Analyse data and use statistics enable the user to understand what the data mean. 
Analysis is one of the most foundational skills in data literacy. By statistics we mean 
simple statistics such as central tendency and dispersion, not more advanced 
techniques like regression or ANOVA. 

• Understanding data displays and representations because data are often graphically depicted, 
in chart, tables, graphs, and other displays. 

• Synthesize diverse data because often times, disparate data are applied to a problem and 
must be pulled together in a coherent manner in order for them to make sense. 

• Summarize data and explain to present the information in a condensed or abstract form 
to aid in explanation and understanding. 

• Draw inferences and conclusions from the information analyzed during the inquiry 
process. 

• Consider impact, consequences, intended and unintended because the outcome of the inquiry 
process and decision is an impact on the circumstances that has consequences which 
can be either expected or unexpected. (Only from the experts’ definitions.) 

Transformation of Data to Implementation 

The transformation of data to implementation is essentially about taking data and using them to 
inform instruction and classroom practice. Some have called this instructional decision making 
(Means, Chen, DeBarger, & Padilla, 2011) or pedagogical data literacy (Mandinach, 2012). This is a 
set of skills not raised by the experts. 

• Pedagogical data literacy, instructional adjustments , pedagogical content knowledge ; that is 
knowing how to apply data to inform instructional decisions, making adjustments to 
classroom practices, and knowing what instructional actions are appropriate given 
the information gained from examining data. 

• Diagnose and monitor to determine students’ learning strengths and weakness and 
monitor performance over time. (Only from the licensure documents.) 

• Plan , design , and guide instruction based on the examination of data. 

• Adapt, modify, individualize, and differentiate instructional steps based on the 
examination of data. (Only from the licensure documents.) 

• Adjust and apply instruction based on the application of information culled from data 
analyses. (Only from the licensure documents.) 

• Transforming data into an implemented decision involves knowing how to take the data and 
understand their ramifications for instruction, then determining and making a 
decision about what instructional actions are appropriate for the given situation. 
(Only from the licensure documents.) 


SUMMARY OF THE FINDINGS, CAVEATS, AND CONSIDERATIONS 

States vary in the extent to which data literacy skills are incorporated explicitly into their licensure 
documents. Some states note many different skills as required, whereas other states outline only a 
few or no skills. Some states are explicit in the degree to which their requirements outline specific 
skills as opposed to making more general statements in which skills are inferred. A small number of 
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states actually have data standards or lay out developmental continua that specify how a skill might 
be exhibited by a novice teacher as opposed to an expert teacher. More states focus on what the 
field would consider assessment literacy than data literacy. This is not a surprising in finding 
because the data that teachers most often use are assessment results. 

The licensure requirements are written in very general terms that may or may not be helpful to 
schools of education, teacher candidates, and other stakeholders in terms of planning actionable 
steps. They certainly are not written with sufficient specificity for the measurement of data literacy 
or even for planning a stand-alone course or how to integrate data-related concepts into existing 
courses. Some statements are directly related to data, whereas for others the relationship to data 
must be inferred. Take for example the following statement extracted from one state’s 
documentation: 

The teacher understands and uses multiple methods of assessment and other student data to 
engage learners in their growth, to monitor learner progress, and to guide the teacher’s and 
learner’s decision making. 

There are several key emphases here. A first is the purpose for the data use, for growth, for learner 
progress, and for decision making. A second is that not just teachers, but also students, are seen as 
decision makers. The objectives are quite clear. The skills are more general. The teacher is to “use” 
“multiple methods” of “assessments” as well as “monitor” and “guide”. The statement does not say 
what the other student data might be. Are they performance data? Are they qualitative data? A 
sub-requirement is to: 

Use assessment results to provide students and parents with timely, helpful, and accurate 
feedback on their progress toward achievement goals. 

In this statement, “communication” and the “involvement of other stakeholders” are emphasized. 
Take these examples of differing specificity and explicitness: 

Understand data practices. 

Teachers read and interpret data and use this analysis to differentiate learning for and tailor 
instructional goals to individual students. 

Analyze data and solve problems using data presented in histograms, bar graphs, circle 
graphs, pictographs, tables, and charts. 

Identify how the presentation of data can lead to different or inappropriate interpretations. 

The first statement says nothing that could be useful to a candidate or an institution, yet it does 
mention data. The reader is left to interpret what is meant by data practices. The statement simply 
is not informative in any constructive way. The second statement is packed with valuable 
information. Even though the first part of the statement is quite general (read and interpret data), 
the use is specific and explicit (the analyses should be used to help the teacher differentiate, 
individualize, and tailor instruction). The third statement introduces yet another factor, solving 
problems. This means that there is an explicit purpose, issue, or problem to which data analysis 
should be applied. Further, the teacher must understand how to read and interpret the information 
in the graphical representations of data. The second part of this statement can help guide topics 
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covered in a teacher preparation course. It can be translated into instruction for and assessment of a 
teacher candidate. It is a skill that is demonstrable. The final statement crosses the boundaries of a 
number of skills, although not explicitly noted, with the inclusion of"inappropriate conclusions”. 
The skill could be to make inferences. It could be to draw conclusions. It pertains to the ethics of 
data use. It pertains to the inquiry cycle and the translation of data to implementation and to the 
examination or interpretation of a decision. This is a highly complex statement that requires more 
than a passing glance. Our analyses yielded in essence word counts. That is, did a document 
contain a particular term like "infer”, "interpret”, "analyze”, or "plan”. What it has failed to 
accomplish, because of the weight of the task, is to dig deeply into the requisite skills and spend the 
time reading between the lines of the documentation by making inferences about meaning. 


These examples are intended to stimulate thought about the complexity of the data we have 
presented and the challenges to making meaningful interpretations of the findings. These examples 
also are intended to illustrate the complexities in accurately and appropriately capturing the nuances 
among the statements found in the licensure documents. Our data are telling about which states 
focus on data and data literacy, those that emphasize assessment and assessment literacy, and those 
where any treatment of the construct may be lacking. We have presented here a cursory, yet labor 
intensive examination of the licensure requirement through a lens that scans for the terms "data” 
and "assessment”. The data that have been yielded by this scan are informative and interesting. Yet, 
because of the shear wealth of information contained in these thousands of pages of documentation, 
the analyses may be only surface level, requiring more intensive examination and analysis. 

The final statement above is an apt example. It would have been categorized as presentation and 
interpretation because of the coding. What it would not have yielded was drawing conclusions, or 
even, as the experts noted as a skill, understanding the accuracy or appropriateness of certain phases 
of the inquiry cycle. This may fall under general aspects of the inquiry cycle, data properties, general 
data use, procedural skills, or the transformation of data. It may involve ethics. It may be another 
overarching component. It crosses a number of components of data literacy. Yet the deep meaning 
of the statement and the nuances would be lost in the analytic process without much deeper 
examination, analysis, and interpretation. Such depth simply is not possible in the current analyses. 

There are a few high-level messages that can be drawn from our results. Just because a state 
includes a large number of statements about data in their documentation does not mean that what 
they are doing is of high quality or well aligned with DLFT. Intent is part of the issue; 
implementation is another part. Second, some skills are noted in many states’ documentation 
whereas others are noted by only a few states. This finding says something about the importance of 
the skill but, not does not provide the complete picture. It is a glass half full/half empty situation. 
Take for example "use multiple sources” which 39 states identified. Using multiple sources of data 
is a well-known and fundamental concept in data use as well as in assessment. Why then did 16 
states fail to include the skill. Is it that they don’t feel it is important? Is it because they feel it is an 
obvious skill? In contrast, take for example "challenge assumptions”, "synthesize”, "organize”, and 
"summarize”. These skills were noted by only a small number of states, yet they are critical skills in 
data literacy. Why did they not rise to the level of importance to be included in the requirements? 
What the data and this interpretation indicate is that shear quantity does not directly translate to 
quality and that quality is difficult to operationalize and highly nuanced. It also means that specific 
statements in the requirements must be carefully crafted with explicitness and alignment to specific 
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skills. The requirements then can be more easily translated into courses, assessments, and 
understanding of the components that comprise DLFT. 

Finding and Understanding the Requirements 

A number of caveats and limitations must be raised about the licensure documentation and their 
analyses. First, the documents were neither easy to locate or transparent. One would assume that a 
state licensing agency would make such documents readily available, easy to locate, and quite 
straightforward. This was not the case. For some states, the process was simple. Yet for many 
states, the process was incredibly arduous. We had to enlist the help of representatives from 
NASDTEC to direct us to the correct URL or even to an individual in a state who could provide us 
with the appropriate documents. In some instances, the URLs had changed or expired and we hit a 
deadend. In a few instances, even having direct conversations with the individuals in charge of 
licensure yielded sites that provided no information or the wrong information. We had been warned 
that, even if we did find documentation, we might have to triangulate among sources. That is, some 
documents might pertain to programmatic requirements, while others might be for the assessment 
of teacher candidates, and yet others geared to the individual candidate. We began to invoke an 
informal test of ease of access to the appropriate information, which we referred to as the click test; 
that is, how many mouse clicks did it take per state to finally identify the correct documents. For an 
exalted few, only one click was needed to hit the target. However, for the vast majority, the process 
took several clicks. And for perhaps ten states, the process led to many deadends, wrong 
documents, or no documents, requiring hours of search by two experienced researchers. 

The take away message here is that such documentation should be easily accessible, readily 
searchable, and transparent once the requirements have been found. This is a general comment 
about the documentation, not about the study. Teacher candidates, schools of education, and other 
interested stakeholders need these documents and they must be in understandable form. 

The messaging in the documents is an entirely different issue. As can be seen in Appendix A, the 
general characteristics of the documents vary tremendously in terms of different rating criteria 
pertaining to data literacy and more generally. Some documents were cursory at best, containing a 
superficial statements about what teacher candidates need to know. Some documents did not even 
address skills and knowledge. Other documents contained pages of specific requirements, identified 
by intended content area and grade level. Still other states opted for not developing their own 
requirements and standards. Instead, they invoked the InTASC standards or used Danielson’s 
framework. In a few instances of those using InTASC, the states embellished the requirements with 
a few of their own. In one or two of the InTASC states, the knowledge that they use these 
standards was only discovered by speaking to an agency representative, not finding the information 
on a website. Thus, both accessibility and messaging are issues. Finally, in once case (WY), there 
are no documents that outline the skills that teacher candidates need, not just about data use but 
more generally. 

Each state certainly must customize their requirements according to their needs, but the differences 
are staggering and perhaps present major challenges to candidates who are trying to understand what 
it is they need to know and demonstrate to become licensed in one state, with no comparability in 
neighboring states where they might also consider working. 

The DLFT Requirements 
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Caveats specific to the data literacy requirements must be considered. First, some states have such 
little information that “data” is not even mentioned in their documentation. Second, “data” and 
“information” might be used interchangeably. However, searching on “information” yields a very 
different pathway that is not entirely relevant to the objective of this work. Third and certainly not 
unexpected, is the conflating of data and assessment. Even if there is no mention of “data”, there 
may relevant requirements that fall under “assessment” because the majority of the data teachers use 
fall under the category of assessment data. Very few states, if any, considered non-assessment data 
to be relevant or within the requirements. At least one state referred to “other data” but these was 
no information about what these data might be. This is a major issue in the field that is being 
brought to the attention of policymakers and they may not even grasp the difference (see CCSSO, 
2012 as evidence of this problem). People think immediately that the term “data” is synonomous 
with assessment results. People are confused about the differences between assessment literacy and 
data literacy (Mandinach & Gummer, 2012, 2013a). We see assessment literacy as a component of 
data literacy because a teacher must be armed with many sources and kinds of data in order to 
obtain a comprehensive picture of a student. These data include much more than simply assessment 
results. They include behavioral, attendance, demographic and personal, health, welfare, attitudinal, 
emotional, and transportation data. Additionally, not all data are quantitative; some are qualitative. 
As can be seen in the alignment work, the licensure documents do not even recognize qualitative 
data, whereas the experts do. The requirements need to reflect the broadest range of data and the 
broadest possible definition. 

Another caveat pertains to our search procedure to locate the data literacy skills. We searched on 
only two terms, data and assessment. In doing so, we probably captured the vast majority of the 
relevant statements in the documents. However, it is entirely possible that some statements did not 
meet those criteria. For statements whose documents are one to a few pages long, it was simple to 
read everything. But for some states, we were confronted with hundreds of pages and multiple and 
often redundant pages. It was not time-effective to read every single word. Therefore, a small 
number of relevant statements may have been missed. Correspondingly and as noted above, a 
substantial number of hours would be required to carry out an in-depth analysis that carefully 
interpreted every nuanced statement. 

Two caveats pertain to the terminology found in the licensure documents. First, different states use 
different terminology but they may be intended to be synonomous or similar. We need to invoke 
some interpretation to discern what terms were being used similarly or differently. An example is 
the category that now includes “access”, “retrieve”, “find”, and “locate”. These terms were found in 
different across states and sometimes even within a state. We needed to make professional and 
interpretive judgments about the meaning and degree of similarity, without which there wourl have 
been perhaps twice or three times as many individual skills in the matrix. Second, the identification 
of the general characteristic categories was intended to capture the range of diversity among the 
documents that might differentiate the requirements. Some states laid out specific skills, whereas 
others did not. Some states talked about data; some did not. Some states included a data standard 
yet even that criterion can be misleading. In one instance, there was a specific data standard yet 
there was little mention of data per se. Some documents contained a lot of statements that were 
considered relevant but were not completely on target with respect to data literacy. Other states may 
have had relatively little information, but what was written was right on target. The reader needs to 
examine the entire picture of a state in order to fully understand the composite. That is why we 
have presented by state analyses as well as the by skill or outcome analyses. Further, there needs to 
be a certain level of interpretation about each state and skill. 
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INTREPRETIVE COMMENTS AND CONCLUDING THOUGHTS 


Interpreting the Results 

Examining the data from the licensure documents requires more than straight analytics. It requires 
examining patterns and trends within and across states. It requires examining the skills and 
outcomes in terms of not just how many states require them, but how they are embedded and 
integrated more generally. It is clear that some states are making great strides into requiring data 
literacy, yet if we go simply by the numbers, they look acceptable not extraordinary. Take for 
example Ohio, North Carolina, Virginia, and Utah. Their numbers do not adequately reflect the 
extent to which data literacy is included in their documents. States like Virginia, Kansas, Utah, and 
New Hampshire fair slightly better. Yet, if we abide strictly by the numbers, the states that invoke 
InTASC come out on top and that may well be a misrepresentation of fact. Or, take for example 
Alabama. Their numbers are quite high but experts in the field do not look to them as doing 
anything groundbreaking. 

Another perspective is to consider if states have self-reported having a data standard to the Data 
Quality Campaign in their annual survey. According to the Data Quality Campaign 2 20 states report 
having a requirement for licensing for teachers and 25 have a program requirement. For the most 
part, the data from the DQC analyses confirm that states with requirement are more data-oriented 
than the remaining states. If we compare the states with and without a licensure requirement, a 
larger percent of states (70% vs. 29%) tend to describe data literacy, are more likely to have a data 
standard (25% vs. 10%), have a less strong focus on assessment literacy (65% vs. 77%), and are less 
likely to not mention data (15% vs. 26%). These data provide some level of validation of the 
criteria. Similar patterns occur for the states with and without programmatic requirements. The 
states without either licensure or programmatic requirements also are less data-oriented. 

Yet the data do not tell all. There are a number of anomalies found in the analyses. Data standards 
appear in the documentation of two states among the non-requirement states. These are Kansas 
and South Dakota. Kansas is one of the few outliers here. They report having neither licensure nor 
programmatic requirements, yet their licensure documents are strongly oriented toward data literacy. 
Kansas is an interesting example because there is substantial leadership around data and there have 
been efforts to provide in-service training to districts and schools. Given that emphasis, having a 
data standard makes sense, yet they failed to report having one to the DQC. South Dakota appears 
data-oriented by virtue of using the Danielson framework. Conversely, there are states where the 
emphasis on data is barely apparent, yet the documentation yielded a substantial number of skills. 
These are states such as Alabama and Nevada, by virtue of InTASC. 

The District of Columbia entitled a section “Data and Assessment” in their documentation, yet the 
term “data” is used in a limited manner. Yet, the documentation does include “make decisions”. It 
exclusively refers to assessment. The number of skills (10) reflects this emphasis. A state like North 
Carolina has a document devoted to defining data literacy on their website (North Carolina 
Department of Public Instruction, 2013). However, if we look at the number of skills listed in the 
actual licensure documents, only 16 skills appear. It is therefore difficult to say what might be an 
acceptable number of skills to adequately reflect the construct. What would be an appropriate 
threshold? Are there certain skills that may be deemed more critical in terms of the standards than 


2 Data provided to the author by DQC staff. 
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other skills? That is, should there be a ranking order of importance for skills? These are difficult 
questions. 

Certainly some of the most frequently mentioned skills are critical. One would predict that skills 
such as “assess”, “collaborate”, “plan”, “evaluate”, “monitor”, “communicate”, “analyze”, “multiple 
sources”, and “collect” should rise to the top. Teachers need to plan instruction. They need to 
monitor performance. Communicating about performance is important. And assess is what all 
teachers do. But even for these most frequent skills, we must question why roughly half or more of 
the states fail to mention them. What is happening in the states that do not include these skills? 
What about skills that appear infrequently or not at all in the licensure documents, yet the experts 
clearly believe that they are essential components of data literacy? 

Concluding Thoughts 

It is clear that the licensure documents provide some, but incomplete guidelines about integrating 
data literacy skills into teacher preparation programs. Heretofore, definitions of data literacy applied 
to teaching were general and not particularly helpful for crafting requirements or even developing 
courses. We have posited a definition that hopefully will fill the gap. We have also identified an 
extensive set of skills and knowledge that can inform future licensure and course development. That 
is our hope. 

It also is clear from our analyses that defining data literacy and its component skills has been a 
challenging process. Many skills have been identified both in the licensure documents and by the 
experts. There is substantial overlap, yet the experts focused on the fundamental principles of data- 
driven decision making, whereas the licensure documents included skills that transform data into 
instructional action. Taking a high-level perspective, the several global constructs can help to define 
data literacy. In contrast, the specific skills can serve as essential information about what needs to 
be included in courses that pertain to data use. They can serve as a roadmap for the development of 
courses on data-driven decision making or the integration of data concepts into existing courses for 
pre-service or graduate education courses. 

The inclusion of data concepts into the licensure documents is not an assurance that data-driven 
decision making is being taught in schools of education. The documents serve as a guide for skills 
and knowledge that should be addressed. The question remains if and how those standards or 
statements are translated into action in courses within each state. Obviously there are differences 
across states. Some states have meager inclusions of data-related statements, whereas others address 
data in a substantial manner. Even though the documentation may be scant, there may be activity in 
the schools of education. Conversely and quite likely, in some of the states where there is he 
substantial documentation, activity around data literacy may be nothing more than lip-service. The 
key here is to ascertain in a reliable and valid manner, the extent to which the licensure requirements 
are impactful in schools of education. There is one form of validation for our coding scheme; the 
Data Quality Campaign’s (2012) annual survey Action 9 and the soon to be released 2013 survey 
results. But there are measurement problem with this index. The survey is self-report and may or 
may not have been completed by a knowledgeable individual about licensure within each state. 

Even if this survey were completed accurately to reflect the licensure documentations, there still is 
no guarantee of actual practice. 

Another issue is the level at which the licensure documentation should be written to have maximum 
impact on the course offerings in schools of education. The language found in the licensure 
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documents is necessarily quite general. They provide a high-level view of what skills and knowledge 
teacher candidates must demonstrate, if they provide the specifics at all. The range of generality was 
quite large. Getting down into the weeds and noting specific, requisite skills with great specificity 
may be too much programmatically for a school of education to accommodate in their courses. 
However, this level of specificity can help professors in pre-service programs, professional 
development providers, and technical assistance programs to include the myriad of skills or a subset 
of them into courses and training. It can also potentially stimulate the development of 
comprehensive course content or training materials that can be used in a variety of teacher 
preparation contexts. So a question remains as to the level of specificity at which the licensure 
documents should be written to convey the kind of actionable information on which schools of 
education can modify curricula, yet also convey the needed specificity about the construct. 

The definition of DLFT and the identification of the component skills and knowledge are two steps 
in the arduous process of helping teachers to become facile with data. Perhaps a more major issue is 
affecting change in institutions of higher education. How to include data literacy for teaching into 
schools of education is a complex and systemic issue (Mandinach & Gummer, 2013b). Lets 
consider the key variables in the equation, their roles, and the constraints place on them. Schools of 
education are at the center of the issue. And the question is how to facilitate their inclusion of data- 
driven concepts into existing courses or stand-alone data courses. The situation is very much like 
the old joke — How many psychologists does it take to change a light bulb? Only one, but the light 
bulb must want to change. Like the light bulb, schools of education must want to change; that is, 
they must want to take on the role of helping to build educators’ capacity to use data. 

Deans have told us that they want to do this, but they have encountered several obstacles. First, 
they have no flexibility in their curriculum to offer a data course. To that problem, we suggest 
integrating data into existing courses. Second, they have no one capable of teaching data courses. 
One solution may be the development of virtual materials designed by experts that professors can 
integrate into their courses. Third, data use is not sufficiently important to which a faculty slot 
might be allocated. The virtual solution again might be viable. Bringing in a local district’s data 
expert as an adjunct might be possible. Contracting with some of the known professional 
development or technical assistance providers might be another option. School districts typically 
use a turnkey model in which principals are trained and they then train the faculty. Using this 
model, it may be possible to bring in an expert to help schools of education faculty recognize where 
data concepts can be integrated into existing courses or how the courses can be modified to 
accommodate the topic. It is worth repeating that the schools of education must want to change. 
We must recognize, however, that institutions of higher education are notorious for resisting change 
and change comes slowly. 

Further, schools of education do not function in isolation. There are many other influences on 
them. Currently, there is increasing pressure for the evaluation of teachers once they have left their 
preparation programs. Teachers’ subsequent performance (or we should say, their students’ 
performance) is seen as a reflection of the preparation programs. This trend brings into the 
equation at least three other variables: the credentialing or licensure agencies; testing organizations; 
and local school districts. Before we describe the roles of these entities, let us present a metaphor 
developed by the Data Quality Campaign (2013). They espouse that data should be used as a 
flashlight to guide continuous improvement, rather than a hammer for accountability and 
compliance. Herein lies the challenge for two of the agencies. The licensure agencies can tighten 
their requirements and schools of education can decide whether or not to respond. The more 
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stringent the requirements become, the more the pendulum swings toward the hammer. If testing 
organizations, such as Educational Testing Service, decide to include data literacy on the Praxis 
(which they are currently discussing), the hammer becomes a sledgehammer, one that will come 
down hard on candidates that fail to exhibit data skills. Local schools districts can also apply 
pressure for change to schools of education by requiring that applicants for teaching positions 
demonstrate their capacity to use data. This is happening in some locations for candidates for 
principal positions (Long, Rivas, Light, & Mandinach, 2008). If candidates are rejected, that reflects 
back on the school of education. This is also a hammer. 

What is needed is the flashlight to illuminate the path to change and continuous improvement. 

First, the schools of education must recognize that they need to integrate data into course content 
and practica. Second, they must develop a strategy for how the change will occur. They know the 
distal goal, but must take proximal steps toward achieving the objective. Third, they must consult 
with a variety of sources that can help them in this change process. This might include institutions 
that have already introduced data into courses and practice. It might also include bringing in expert 
researchers in the field, professional developers, in-service providers from local school districts, and 
other who might provide guidance and models for successful integration and course content. 
Fourth, they might consider working with the credentialing and licensure agency in their state to 
align the evolving content with the requirements. Finally, they might consult with test developers at 
Educational Testing Service about their intent to include data literacy in Praxis to better understand 
how they are conceptualizing the construct. That knowledge can be reflected back on the design of 
courses in the school of education. MOOC’s might be another viable solution. The provision of 
such a broad-based course offering could provide data-driven courses to many schools of education 
throughout the country. 

There is no getting away from both the complexity of the construct and the systemic nature of the 
change process. Data-driven decision making is not a passing fad, much to the dismay of many 
educators who are fearful of the changes in practice and the ramifications of using data for 
compliance, accountability, and evaluative purposes. Those are the hammers or cudgels. Educators 
do not need more of them. If the change process is managed in a logical, intelligent, and humane 
manner, it can function with sensitivity as a beacon of light to help all educators function more 
effectively. 
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Appendix A 

General Characteristics by State 


State 

Amt. 

Data 

Assess 

Sp. 

Skills 

Specif! ci 
ty 

InTASC 

Dev. 

Cont. 

Data 

Std. 

CDF 

Data 

Lit. 

Assess 

Lit. 

AL 

H 

X 

X 

X 
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XX 


CA 

H 

X 

X 

X 
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X 

/ 

DE 

H 

X 

X 

X 

H 

X 
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X 

FL 

H 

X 

X 

X 

H 





X 

X 

GA 

M 

X 

X 

X 
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/ 

/ 
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/ 

X 

X 
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X 
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X 

ME 

M 

NO 

X 

X 
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X 

MD 
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/ 

X 

X 
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X 

MN 
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/ 

X 

X 

H 
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L 

NO 

X 

NO 

L 
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L 
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NO 

/ 
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XX 

XX 
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X 
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X 
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X 

X 
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X 
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H 

X 

X 

X 
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XX 

XX 


XX 
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H 

X 

X 

X 
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X 
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XX 

X 

XX 

SC 
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X 

X 

X 

H 

X 




XX 

X 

X 

UT 
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X 
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XX 

X 
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VA 

H 

X 

X 

X 
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XX 

X 

XXX 
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Explanation of columns 

Amt. — Amount of data-related information found in the documentation. 

Data — The extent to which the documentation mentions data explicitly. 

Assess — The extent to which the documentation mentions assessment explicitly. 
Sp. Skills — Do the documents mention specific data-related skills. 

Specificity — The degree of specificity of the skills. 

InTASC — Does the state invoke the InTASC standards. 

Dev. Cont. — Does the state present a developmental continuum for a skill. 

CDF — Does the state invoke Charlotte Danielson’s framework. 

Data Lit. — Does the documentation really address data literacy. 

Assess. Lit — Does the documentation really address assessment literacy. 
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Appendix B 

List of Skills and Knowledge from Experts’ Definitions 

Identify problems of practice 

Synthesize diverse data/information/assimilate 

Engage in inquiry cycle 

Identifying the right data/select data/qualitative and quantitative/evaluate 
applicability/understanding utility of data 

Understanding data quality and limitations — accuracy, completeness 

Understanding how to use the findings 

Understanding the purposes of different data 

Framing questions 

Using multiple sources of data 

Using formative and summative assessments/knowing which assessments/benchmarks 

Knowing where to find the right data/data location 

Ability to assess patterns/trends 

Knowing the research that can inform the issue 

Summarizing data/Explain 

Communicating information/findings 

Ability to use technology to support data use 

Collect data/store 

Analyze data 

Analyze different levels of data — cohort, course, grade 

Drilling down to different layers of data — aggregate, disaggregate, strand, item 

Rethinking with new information 

Collaboration 

Instructional decision making/pedagogical data literacy/instructional adjustments 
Interpretation 

Data comprehension [data displays and representations/reports/presentations/longitudinal, cross- 
sectional] 

Unpacking information 
Ability to solve problems 

Making meaning/identify and critique meaning 

Marshal facts/Support or refute 

Use of statistics 

Use of quantitative evidence 

Use qualitative data 

Draw inference 

Knowledge of assessments/psychometrics 
Application of interpretations 
Organizing 

Understanding how to involve stakeholders 
Evaluate outcomes/scrutinizing results 
Action that leads to change 

Transforming data into information and knowledge and actionable decisions 
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Translating knowledge into appropriate responses/implement decision/draw conclusions 
Ability to access data 

Ability to evaluate success of proposed solution/feedback loop 

Ability to understand the data-driven process 

Understand the context in which decision is being made 

Probe for causality 

Generate hypotheses 

Test assumptions/hypotheses 

Manipulate data 

Using habits of mind 

Think critically 

Critique arguments 

Considers impact, consequences - intended or unintended 

Pedagogical content knowledge 

Administrative knowledge 

Organize data 

Prioritize data 

Determines additional next steps 
Communicate findings 
Understanding what data are not actionable 
Troubleshooting problematic data 
Ethics of data use 

Understanding scaled scores, percentiles, performance levels 
Belief in data 
Identifying purpose 

Developing a sound design of assessments 
Involve students 

Metacognitive aspects/knowing or nor knowing about data 
Generating data 
Use data 

Instructional data literacy = statistics literacy + assessment literacy + analysis literacy 
Content [what] and Performance Components [how much knowledge and skills] 
Understanding the difference along a continuum from basic skills in data literacy to complete 
fluency — level of ability necessary to be considered data literate 

Awareness of knowledge/skill — Application of knowledge/skill — Mastery of knowledge/skill 
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Appendix C 

Number of Skills Per State 


State 

Number of 
Skills 

State 

Number of 
Skills 

State 

Number of 
Skills 

AK 

9 

AL 

35 

AR 

50 

AZ 

49 

CA 

25 

CO 

11 

CT 

22 

DC 

10 

DE 

49 

FL 

28 

GA 

16 

HI 

11 

IA 

9 

ID 

24 

IL 

36 

IN 

20 

KS 

34 

KY 

24 

LA 

4 

MA 

7 

MD 

12 

ME 

10 

MI 

23 

MN 

23 

MO 

8 

MS 

6 

MT 

5 

NC 

25 

ND 

52 

NE 

8 

NH 

34 

NJ 

19 

NM 

10 

NV 

50 

NY 

29 

OH 

29 

OK 

22 

OR 

24 

PA 

15 

RI 

19 

SC 

52 

SD 

15 

TN 

18 

TX 

17 

UT 

28 

VA 

20 

VT 

17 

WA 

6 

WI 

3 

wv 

21 

WY 

0 
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Appendix D 

Number of States Per Skill 


Skill 

No. of 
States 

Skill 

No. of 
States 

Skill 

No. of 
States 

Assess 

43 

Modify 

28 

Collaborate 

28 

Involve 

stakeholders 

36 

Make decisions 

28 

Identify/Select 

20 

Adapt 

21 

Plan 

39 

Evaluate 

33 

Use technology 

29 

Monitor 

36 

Displays/Representations 

14 

Inquiry 

15 

Reflect 

26 

Question 

9 

Challenge 

Assumptions 

6 

Communicate 

34 

Manage 

12 

Verify 

6 

Document/Review 

31 

Examine 

8 

Feedback 

27 

Differentiate 

21 

Self-assess 

19 

Adjust 

26 

Acce s s / Retrieve / Find 

13 

Analyze 

33 

Interpret 

26 

Apply 

10 

Multiple sources 

39 

Implement 

18 

Use 

37 

Generate 

8 

Infer 

7 

Organize 

12 

Process 

8 

Review 

7 

Integrate 

6 

Develop assessments 

16 

Solve problems 

12 

Design/guide 

26 

Diagnose 

18 

Summarize 

4 

Predict/hypothesize/ 

postulate 

5 

Act/enact 

3 

Ethics 

18 

Research 

19 

Statistics 

8 

Collect/gather 

32 

Draw Conclusions 

11 

Individualize 

8 

Synthesize 

13 

Disaggregate / group 
differences 

13 

Goal setting 

11 

Different data 

23 

Implications/impact 

16 

Patterns/ trends 

9 

Manipulate 

2 

Data quality 

12 
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Appendix E 

Number of States Per Outcome 


Outcome 

Number of States 

Outcome 

Number of States 

Instruction 

46 

Guidance 

8 

Performance 

34 

Program effectiveness 

6 

Assessment 

48 

Local standards 

14 

Student learning 

40 

Curricular goals 

28 

Student progress 

41 

Readiness 

9 

Patterns and trends 

9 

Understanding 

11 

Accountability 

7 

Achievement 

21 

Learning gains 

7 

Student needs 

32 

Abilities/ skills 

17 

Growth 

20 

Teaching 

22 

Outcomes 

16 
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Appendix F 

Alignment Matrix 


Components 


Skills 


Superordinate Construct 


• Engage in inquiry cycle 

• Communication 

• Transforming data to information 
to actionable knowledge 

Inquiry Process 


• Identify problems of 
practice 

• Generate hypotheses 

• Frame questions 

• Probe for causality 

• Understand context for the 
decision 

• Involve stakeholders, including 


• Test assumptions 
(early) 

• Evaluate outcomes / 
scrutinize results / 

parents 

• Determine next steps 


• Test hypothesis (late) 

review / feedback 
loop 


Habits of Mine 



• Think 
critically 

• Belief in data 

•Collaboration 

Data Use - General 


• Use of data 

• Ethics of data use 


Data Quality 


• Understanding data 
quality — 

accuracy, completeness 
/ verify 

• Understanding 
problematic data 

• Knowledge of assessments / 
psychometrics 

Data Properties 


• Understanding the 
purposes of different 
data 

• Identify and evaluate 
the right data 

• Use qualitative data 

• Use quantitative data 


• Understand what data 

• Use formative and 

• Use multiple measures / sources 


are not applicable 

summative 

of data 



assessments 



• Use research evidence 



Data Use Procedural Skills 


• Use technologies to 
support data use 

• Ability to find, locate, 
access, retrieve data 

• Generate 
data 

• Prioritize 
data 


• Collect, gather, store 
data 

• Organize data 

• Manipulate data 


• Examine data 

• Manage data 

• Reflect on data 


• Process data 

• Integrate data 

• Review / document 


• Develop sound 
assessments 

• Drilling down, aggregating, 
disaggregating, strand, items 

Transform Data to Information 
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• Transform data to • Analyze data / use • Interpret — make meaning 

information to statistics • Understanding data displays and 

actionable knowledge • Assess patterns, representations 

trends 

• Synthesize diverse data • Summarize data / • Draw inferences / conclusions 

explain 

• Consider impact, consequences, intended and 
unintended 

Transform Data to Implementation 


• PDL, instructional • Diagnose / monitor • Adapt / modify / individualize / 

adjustments / • Plan / design/ guide differentiate 

pedagogical content • Adjust / apply 

knowledge 

• Transforming data into an implemented decision 

Communication 


• 

Communication 
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